This invention relates in general to electronic data handling devices and, more particularly, to interactive electronic data processing and storage devices having integrated data displays. Specifically, the invention is directed to a portable, interactive electro-optic data input/output, processing, storage, and display device responsive to hand printed text and hand drawn graphics, as well as hand entered commands, input directly onto the display surface, for providing an electronic writing and drawing slate. The device also provides an input/output capability with other information processing equipment, such as a printer, plotter, or computer.
Various display systems are known which incorporate the capability for pointing or writing directly onto a display screen, along with a detector for sensing and encoding the location of the finger of a user or a writing stylus on the display screen so that graphic information can be drawn directly onto a display surface. These display systems include devices utilizing a separate sensing substrate (such as a touch screen), high power (e.g., CRT, light emitting diode, and electroluminescent) devices, and non-emissive displays using display element locations as input points. These display systems have various shortcomings.
Touch screen and other display systems utilizing separate hardware on the display surface for sensing the position of a writing stylus (or finger), such as a dedicated sensing layer, are typified by input resolution lower than desirable for interactive input and processing of detailed graphics. Input registration and parallax present problems for interactive graphics when physically separate sensing and display elements are employed at resolutions desirable for detailed graphics. Limits on miniaturization of sensing hardware and dedicated interconnecting lines for addressing individual sense locations also place restrictions on achievable input resolution. Additionally, fabrication costs and complexity are increased by the need for separate dedicated sensing hardware on the display surface, as well as requirements that the sense locations be interfaced with display hardware for properly associating the sense locations with the appropriate display elements.
CRT's and display systems employing other emissive (e.g., LED and electroluminescent) displays in connection with a light pen do not require auxiliary sensing hardware on the display surface. The need for auxiliary sensing hardware on the display surface is eliminated by providing a light receiving pen which senses light emitted from the display elements. The optical output from emissive display elements detected by the light pen is then used to determine pen position. See U.S. Pat. No. 4,268,826. Since these display systems use emissive display elements, however, they consume more power than desirable for a portable battery-operated device.
Non-emissive mechanical displays incorporating a writing instrument employ mechanical display elements moved through a fluid whenever the state (i.e., "on" or "off") of any of these elements is changed. See U.S. Pat. No. 4,520,357. These mechanical displays require greater operating power than other non-emissive electronic (e.g., liquid crystal) displays. Additionally, the writing instrument changes the state of display elements by moving them physically, while leaving the previously stored state of any touched display element in memory unchanged. Input to memory is effected by detecting the state of each display element after input by the writing instrument, and then comparing the detected state to the state previously stored in memory. The memory can then be updated to reflect any inputted change in state. Since input depends upon changing the state of the display elements, and since the writing instrument can change only "on" display elements to "off," or vice versa, the user is able to effect input only to one of either "on" or "off" display elements at any one time. This is a substantial limitation to interactive word processing and graphics, as well as to other input operations, which desirably accommodate input to arbitrary display element locations (regardless of whether they are "on" or "off" ).
Other non-emissive displays utilizing display elements as input points are also known. These include a thermally addressed smectic liquid crystal device (see "Thermally Addressed Liquid-Crystal Flat Display with Laser Light Pen," Society for Information Display, Digest of Technical Papers, 1985, pages 289-292), as well as an interactive electrophoretic display (see "A Stylus Writable Electrophoretic Display Device," SID Digest, 1979, page 44). The thermally addressed device requires an operating power of 25-35 watts, as well as a full display write time of 10 seconds, making it unsuitable for highly portable and interactive applications. The electrophoretic device requires a pen voltage of over 200 volts to obtain a reasonable writing speed.
The principal problem with the interactive mechanical, the thermally addressed, and the electrophoretic displays, however, is that they all perform their pen input functions by physically toggling the state of display material to be only one of either "on" or "off," then periodically comparing new input from the pen to display states that are stored in memory, and then updating the memory with the new input. Consequently, all of these devices have the problem that pen input cannot be made to arbitrary display locations regardless of whether those locations are "on" or "off." This limitation makes these devices undesirable in a majority of interactive applications that may involve pointing to arbitrary locations on the display. Furthermore, since pen input is accomplished by modifying the state of display elements, the achievable speed of input is dependent upon display fluid and mechanical properties. This causes pen movement limitations, particularly at high resolutions and refresh rates.
Additionally, low power electro-optic displays suitable for displaying text and graphics are known. These displays can be constructed with high pixel densities including surface areas containing many thousands of individually addressable pixel display elements. The compact nature, flat profile, light weight, and low power requirements of some of these displays have contributed at least some degree of portability in such products as laptop personal computers which embody a flat panel information display with sufficient pixel density to display graphics and with a large enough active surface area to display multiple lines of text. Unfortunately, these displays are typically associated with additional components, such as keyboards, which detract from the overall compactness and portability and do not accommodate direct hand entry of data onto the display surface. Further problems associated with using known techniques in connection with a high resolution, portable data input/output, processing, storage, and display device will become apparent in view of the following observations.
Low power consumption is essential, in the context of portability, to facilitate operation for long periods between battery replacements, as well as the use of inexpensive and readily obtainable batteries which occupy minimal space. Additionally, low power consumption enables the use of other modes of compact and portable power, such as solar cells. Also, low power consumption enables storage of large quantities of information, for longer periods, in low cost, high capacity, volatile memory.
Non-emissive displays (e.g., twisted nematic LCD's) offer very low power consumption and low cost in comparison to other known flat panel devices and can be produced by well known manufacturing processes and with high resolution active matrix addressing for high contrast and wide angle viewing. Among the problems in employing these displays, however, is detection of writing stylus position. By way of contrast, on the one hand, non-emissive displays using display elements as input points (e.g., mechanical displays) avoid the need for auxiliary sensing hardware on the display by employing display elements themselves as pen sense points by providing an input writing instrument which physically toggles the position or state of display elements in a way that is both visible as a change in display state (e.g., "on" to "off"), and detectable by display electrodes (e.g., as a local capacitance change). In order to effect input, these displays compare display element states before writing instrument input (stored in memory) and after writing instrument input. Thereafter, any inputted changes to the display are updated in memory.
On the other hand, emissive display systems eliminate the need for auxiliary sensing hardware by utilizing a light pen with the ability to sense the optical output of emissive display elements themselves. In contrast, a non-emissive display (e.g., twisted nematic LCD) which does not physically switch the state of bistable display elements with pen input does not toggle the display elements for electronic sensing, nor are its display elements light emitting as in emissive displays with pen input.
An additional problem in such a display relates to the speed with which writing stylus input must be sensed in order to accommodate high input resolution without auxiliary sensing hardware on the display. Emissive, mechanical, and other display systems, which use the display elements themselves as pen sense points, perform the sensing function by sampling each pen sense location on a display surface, either in concert with display output, or in a separate cycle, and for each pen sense (display element) location, the presence (or absence) of the writing instrument or light pen is determined. To accommodate a human observer, an interactive display must check all pen sense locations on a display surface for new input, and then update the display for viewing any new input, within a "frame time" period of roughly 30 milliseconds. In this regard, the sampling of display elements in display systems employing display elements as pen sense points is adequate provided there is low enough input resolution that natural pen movement during a frame time is unlikely to traverse more display elements than the number of times each display element location can be sampled for input during a frame time. This is necessary in order to provide sufficiently rapid tracking of arbitrary writing stylus movement. Some known display systems incorporating high power emissive display elements achieve more rapid pen sensing, but only do so at the expense of requiring that added pen sense instrumentality auxiliary to the display itself be fabricated into the display system (e.g., Japanese Patent 58-17247(A), which incorporates a piezoelectric pen sense layer).
In the case of interactive graphics where high pixel resolution (and concomitantly high pen sense element density) is desirable, and given a reasonable pen sense element spacing of, for example, 10 mils, it is conceivable for known displays employing display elements as input points that a writing instrument or light pen could travel from one pen sense element to another in less time than would be possible to sample all pen sense elements on a display. To address this problem, known interactive displays typically incorporate dedicated sensing hardware (e.g., a dedicated sensing layer or an external digitizing system), suffer reduced resolution and/or display area (in order to reduce the number of points sampled in a frame time), or restrict the speed of pen movement.
It is also pointed out that known display systems have heretofore been limited in the amount of information which can be stored internally on electronic memory devices. This limitation is attributable not only to the storage capacity, physical size, power requirements, and cost of available electronic memory, but also by the extent to which the displays can store commonly used symbols, such as text characters and graphic shapes, directly and in a compact and referenceable format, such as ASCII, suitable for word processing and for interface to conventional text processing and printing devices.
Conventionally, typefont text characters, each displayed as a pattern of many individual pixel display elements, can be input to a display via a keyboard device, in a standardized coded format, such as ASCII. This facilitates efficient text storage, word processing, and interface to conventional text processing devices, such as printers, which accept standardized text formats.
However, in order for text characters to be thus input in a compact and referenceable way, the additional instrumentality for text character selection, such as a keyboard, is typically employed. The keyboard must be interfaced to the display and display storage means, which detracts from the overall portability of the display system by requiring a separate structure to accommodate the customary array of fingertip sized keypads required for comprehensive keyboard text input, or by occupying the limited surface area on the display itself to accommodate the required text selection instrumentality, thereby increasing size and weight and/or limiting display area needed for enhanced viewing and editing of full page documents.
If, however, text is input directly onto the display surface via a writing stylus, then hand written textual information can be input, displayed, and stored without requiring text selection instrumentality auxiliary to the display itself. However, substantially more electronic memory is typically required for storing each hand written text character, since each character consists of a multiplicity of pixel display elements, each of which may need to be separately represented in memory, in contrast to text characters represented in a standardized format usually requiring only a single byte for each character stored. For a display input and storage device possessing a notepad or notebook sized display page and which stores hand drawn text or graphics, considerable electronic memory is required for storing more than just a few display pages of information. Even with data compression, such as disclosed in T. S. Huang, et al., "Runlength Encoding and Its Extensions," Symposium on Picture Bandwidth Compression, Massachusetts Institute of Technology, 1969, Gordon and Breach (New York), 1972, the monetary cost of electronic memory severely limits the price/performance, i.e., cost versus storage capacity relationship. Additionally, word processing functions, straightforward with characters represented in a compact and referenceable format, such as ASCII, are much more cumbersome with hand drawn characters. Also, conventional text printing devices are not well adapted to reproduce hand drawn text.
Alternatively, computer terminal devices incorporating a dedicated digitization indicator board equipped with a writing instrument, and utilizing character recognition techniques for input of hand printed commands and text are known. See, e.g., "Micropad Handwriting Terminal," Computer Equipment Review, Vol. 3, No. 1, pages 51-55, and U.S. Pat. No. 4,562,304. These on-line hand print character recognition systems have been found useful in applications where limited amounts of text are routinely input to a computer. For example, these systems are used in lieu of conventional inventory forms or hotel guest sign-in forms.
However, the character recognition techniques associated with these on-line hand print character recognition systems require that adjacent characters be explicitly separated. Accordingly, each character must be drawn within the confines of a prescribed area, or handprint box, printed in grid paper form pre-aligned to a writing panel prior to use. Grid or box entry precludes adjacent characters from overlapping or touching, as is common with natural hand printing.
Another device, the Casio PF-8000, permitting a single character to be printed individually within the confines of an associated digitization panel by means of a user's finger tip, is also known. However, in addition to the above mentioned constraints, this device also prescribes stroke direction (i.e., up or down), as well as the order in which strokes may be drawn when inputting text characters.
In any event, known character recognition techniques provided for on-line hand print character input systems require that characters be printed by prescribed rules which encumber natural hand printing, and consequently make input of lengthy text strings inefficient by comparison to the alternative of keyboard input. Additionally, variations characteristic of natural hand printing, such as stroke overwriting, character skew, serifs, or arbitrarily connected or nonconnected strokes, are restricted or disallowed by currently available on-line hand print character recognition devices. Furthermore, the display surface on these devices, being incapable of independently accepting input from a writing instrument, only accepts data from an associated and separate digitization panel which is itself incapable of independently displaying information, so that the writing instrument cannot be used for directing display processing changes or input to a particular character or location within a body of converted text, displayed on the display surface. Word processing instead requires explicit cursor positioning controls responsive to keyboard commands, mousing, or the digitization panel, the added instrumentality of which detracts from portability and complicates operation.
Less restrictive character recognition techniques than those mentioned above are known (e.g., C. C. Tappert, "Adaptive On-Line Handwriting Recognition," Seventh International Conference on Pattern Recognition (Cat. No. 84CH2046-1), Montreal, Quebec, Canada, Jul. 30-Aug. 2, 1984 (Silver Spring, MD:IEEE Comput. Soc. Press 1984, Vol. 2, page 1004-1007). However, they are computationally more complex and consequently less cost effective and/or slower than are used in current on-line hand print character input systems.
For a highly portable and interactive data input/output, processing, storage, and display device, a text and high resolution graphics capability is needed that can be operated without requiring auxiliary text selection hardware, such as a keyboard, and without consuming available display surface area with keyboard text selection instrumentality auxiliary to the display itself. Additionally, there should be provided an economical and rapid on-line hand print character recognition capability that allows pages of text to be efficiently input, and stored with a minimum of on-board electronic memory devices, and in a form amenable to word processing and data I/O in association with conventional text processing/printing devices.